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BESICOPYAVAIIABIE 



Abstract 



Current research suggests that quality training tor child care providers can 
potentially improve the services ottered to young children. There have been 
several major studies ot child care training and this evaluation is modeled In 
part on these previous evaiuatio.i projects. The Rhode Isiand Child Care 
Training System (RICCTS) was evaluated in terms ot the effectiveness ot the 
training provided. One hundred twenty tive training subjects and 23 
comparison group sut^ects and their supervisors provided data in order to 
answer tour spetitic research questions: 1) Is the training effective in 
improving trainees’s perceived level ot knowledge and skill?; 2) Is the training 
effective in improving the quality ot work iite tor the tranees?; 3) is the 
training effective In enabling the implementation ot trainee created action 
plans?; 4) Are differences in effectiveness evident tor the different types 
training offered (workshops, conterence, and on-going seminars)?. The 
tindings suggest that the RICCTS Is ettective in improving trainees’ level ot 
competence and skill, Is effective in impovlng trainees quality ot work lite, and 
is effective In enabling trainees’ to implement changes in their programs. 

More research Is needed to answer the fourth question because the size ot 
sub-groups was too small to detinitiveiy answer the questton. Implications tor 
training and research are provided. 
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introduction 

The purpose of this study is to evaluate the effectiveness of the Rhode Island 
Child Care Training System (RICCTS) in its efforts to insiire quality programming for 
young children and their families. Past research suggests that the quality of child 
care programs affects children's lives (Kagan, 1994; Galinsky, Shubilla, Wilier, 

Levine, & Daniel, 1994; Vandell & Powers, 1983). The quality of child care 
programs is directly related to the specialized training received by practitioners 
(Morgan, Azer, Costley, Genser, Goodman, Lombardi, & McGinmsey, 1994). The 
vision of the RICCTS is that it will assist in developing a state-wide system of early 
childhood training. The ultimate goal of the RICCTS is to tie all aspects of the state's 
child care and education training into a system that will produce staff who are 
competent in delivering high-quality programs for young children. 

Three justifications exist for evaluating the RICCTS. The first reason to 
conduct this evaluation lies in the importance for evaluating all funded social 
programs to document program accountability, determine program effectiveness, and 
justify future support from funding agencies. The second justification is that the 
RICCTS itself is interested in an evaluation to assess its current effectiveness and 
plan future prograinming. The final justification for this endeavor is the significance 
this study has for the student in terms of contributing to the advancement of program 
research and improved child care. 

It is increasingly the case that social programs, and child care training 
programs in particular (Wolfe, 1994), are employing program evaluations. In fact, it 
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has been shown through previous evaluation research that staff training for child 
care providers is critical in determining overall program quality ( Arnett, 1989; Berk, 
1985; Jorde-Bloom, 1989a; Vandell & Powers, 1983; Wolfe, 1994). 

As a social program, the flICCTS should be accountable for meeting its broad 
and specific goals. According to Adams and Schvaneveldt (1991), program 
evaluation has utility in broadening program effectiveness in that it can provide 
feedback to interventionists who are interested in assuring positive outcomes from 
their efforts. 

The RICCTS board members, as responsive members of the child care 
community, are interested in assessing their program effectiveness. They seek to 
determine whether their mssion statement and goals are being effectively 
implemented. A copy of the mission statement is in Appendix A. 

The establishment of the RICCTS is the culmination of many collaborative 
efforts in the early childhood community. The Training System is funded under a 
grant awarded to Children's Friend and Service by the Department of Human 
Services (DHS). The department, in its role as lead agency, is able to offer this grant 
though funds provided by the Federal Child Care and Development Block Grant. 

The RICCTS developed their specific goals for training based on their broad 
vision for a comprehensive training system that will provide staff who are competent 
in delivering quality programs for young children. The RlCCTS's approach to 
developing specific trainings was also informed in part by the work of the Center of 
Career Development at Wheelock College (Morgan et al. , 1994). The Center found 
that the majority of non-credit trainings are repetitive, the content of most training 
does not address the spectrum of needs in the early childhood field (namely family 



child care), there are inequities in access to both non-credit and credit training, and 
there are few linkages between non-credit and credit training. The Center 
recommends the following for developing a career development system. Create a 
planning body that results in comprehensive long term plans for career development, 
improve access to training for working practitioners, and license individuals as well 
as facilities (Morgan et ai. , 1 994). 

The RICCTS attempts to address the Center's findings in its approach to 
training. They provide training for all aspects of the child care community including 
family child care providers, center-based providers, and directors. The RICCTS also 
offers a variety of types of training including; one-shot two hour workshops, on-going 
8 week training programs, and day long conferences. A variety of topics have been 
addressed in these various trainings such as health and safety, first aid, CPR, parent 
involvement, lead teacher academy, and family day care accreditation. 

Research suggests that different approaches to training yield differences in 
improved services. For instance, evaluations have found that ongoing training is 
more effective than one-shot workshops (Sims, 1990; Jones, 1993). Also, research 
suggests that training programs including discussion, simulation, and application of 
training concepts are more effective than training programs involving only lecture 
(VanderVen, 1993). 

The RICCTS is concerned with providing various types of training for various 
types of needs. They have offered one-day workshops as well as on going training 
to meet the professional and personal demands of the community they are servicing. 
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Th© Training Systom is intGrastGd in dGtGrmining if all of thGSG trainings ar© 
improving programs in th© stat©. They are also interested in using evaluation results 
to shape future trainings, therefore it is important to compare the effectiveness of the 
different types of trainings offered. 

In addition to assessing program effectiveness and providing direction for 
future programs, this study is important because of its potential contribution to the 
program evaluation literature, particularly in early childhood education training. 
Program evaluation can be difficult to administer due to the unique problems that the 
research setting commonly presents (Adams & Schvaneveldt, 1991) and there is 
minimal standardization in the research designs employed. The design employed in 
this evaluation can potentially be used as a model to be built on and revised for 
future program evaluations. Considering the itxreasing call for accountability for 
social interventions, due to funding issues, there will undoubtedly be increasing 
need for program evaluation. Program evaluation could be completed more readily if 
a more comprehensive literature base were available to model an evaluation design. 

Although there is a need for a more evaluation research, there have been 
evaluations of training programs that have been documented in the literature. Past 

4 

research has been corxlucted to assess the effectiveness of different approaches to 
training. Evaluators have used different instruments to measure the effectiveness of 
training. Some have measured trainees' level of perceived competence, and 
trainees' level of satisfaction with work environment (Jorde-Bloom, 1989a). 
Instruments have been developed and standardized to measure both of these areas. 




4 



5 



Another approach to assesfiing the effectiveness of training is to evaluate the 
effectiveness of trainirv^ implementing individualized plans and generally these have 
been measured qualitatively. This study intends to ircorporate all approaches in 
assessing the effectiveness of training. 

Finally, this study is important because it is consistent with the investigator's 
interest in promoting sound social programs. The investigator is concerned with the 
well being of children and families, and therefore is interested in assessing 
interventions that claim to meet this goal. 

Given the justifications for this evaluation, the following specific questions 
were investigated; 1) Is the training offered by the RICCTS effective in increasing 
trainees level of perceived competence and skill?, 2) Is the training effective in 
improving the perception of quality of work life for the trainees'?, 3) Is the training 
effective in enabling trainees to implement changes in their program?, and finally, 4) 
Are there differences in effectiveness for the three types of trainings offered through 
the RICCTS (one-shot workshops, conferences, and on-going training)? 

Method 

Design 

The evaluation of the Rhode Island Child Care Training System used a non- 
equivalent control group design (Cook & Campbell, 1979) and involved multiple 
sources of data (Smith & Glass, 1987). In addition to a comparison group, the 
evaluation design included the collection of longitudinal data. Although this is not a 
true-experiment, the design includes precautions to decrease threats to internal 
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validity including history, instrumentation, pre-post sensitivity, instrumentation, and 
regression (Adams & Schvaneveldt, 1991). Threats to external validity were 
decreased as well. For example, the comparison group received a "treatment" to 
minimize the Hawthorne effect (Huck et al., 1984) 

The evaluation of the RICCTS utilized three outcome variables; (1) trainees' 
level of perceived competence, (2) the perceived quality of work life for the trainees, 
and (3) the implementation of trainee formulated action plans based on the topic of 
training received. Action plans are simply three action based changes that the 
trainees wished to implement based on the specific Iraining they received. These 
outcome variables were assessed from several different perspectives in terms of 
people (trainees, supervisors, and observer [goals]) and in terms of time (pre, post, 
and retrospective). 

Subjects 

The subjects in the experimental, or training group, were self-selected 
volunteers from home day care and center based child care programs from various 
locations in Rhode Island. The training group subjects consisted of all of the 
individuals who attended any of the nine training sessions conducted by the RICCTS 
from October 1994 to February 1995 from which data were collected. There were 
nine different training sessions evaluated in this project, covering five different 
content areas with a total of 125 subjects. All of the training group subjects were 
females, the majority Caucasian, between the ages of 25-34, married, having at least 



some college, with family incomes of approximately $25,000. The majority of the 
training group were child care providers (as opposed to administrators). The mean 
number of years the training group had worked with children was five years. 

The comparison group subjects were randomly selected from a list compiled 
from RICCTS records of child care providers who had not attended RICCTS training. 
There were 23 subjects in the comparison group. Similar to the training group, all of 
the comparison group subjects were females, the majority Caucasian, between the 
ages of 25-34, married, having at least some college, with family incomes of 
approximately $25,000. The majority of the comparison group were child care 
providers. The mean number of years the comparison group had worked with 
children was five years. 

All subjects provided informed consent prior to their participation. A copy of 
the informed consent document is included in Appendix B. 

Instrumentation 

The design of this study utilized several approaches for measuring program 
improvements; feedback from trainees regarding changes in their perceived 
knowledge and skills; feedback from supervisors attesting to changes in trainee 
behavior or attitudes; arxi independent observations by an outside party assessing 
actual on the job behavior post training. Multiple perspectives were used to increase 
the reliability and validity of results by decreasing the possibility of bias. Copies of all 
instruments used in this study are irx;luded in Appendix C. 



Demographic Information. All subjects completed a Trainee Characteristics 



Survey. This survey asks for demographic data on 16 characteristics such as 
education, race, income, occupational experience, present position, size and type of 
program, and past training received within and outside of the RICCTS. 

Level of Perceived Competence. The trainees' level of perceived 
competerce was measured using a modified version of the Training Needs 
Assessment Survey (TNAS; Bloom, Sheerer, Richard, & Britz,1991). TheTNAS 
assesses level of perceived competence in knowledge and skill areas related to the 
care and education of young children. Although this instrument was created to 
assess director training, it contains items relevant to a broad range of child care 
providers. The original instrument had 28 items. For the purposes of this study, 15 
items that were too specific and related solely to director training were eliminated. 
The remaining 13 knowledge and skill areas assessed by the revised TNAS are 

clustered into two areas; 

1. Personal/professional self-knowledge 

The four items in this area assess knowledge of one's learning and teaching 

style and how to apply or adapt that style as situationally appropriate. 

2. Child development and early childhood programming 

These nine items assess knowledge of developmentally appropriate 

principles, and skill in applying developmentally appropriate practices. 

This instrument provides scores for these two areas and a combined score for 
the 13 total items. 

On a 5-point Likert-type scale, respondents were asked lo indicate their level 
of knowledge or skill on each of the 13 items from "no knowledge in this area" (1) to 
“extremely knowledgeable" in this area (5). fheir supervisors were asked to fill out a 
supervisor version (S-TNAS) that contained the same items. The total possible 
range of scores for both surveys is 13 to 65. Copies of the TNAS and S-TNAS are 
included in Appendix B. 
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Content validity of the TNAS was established in a previous study assessing 
the training needs of 990 center directors in Illinois (Jorde-Bloom, 1989b) and for 
leadership training for directors (Jorde-Bloom & Sheerer, 1992). The TNAS was 
found to have a coefficient for internal consistency of .95 on the pretest and .96 on 
the posttest using Cronbach's alpha (Jorde-Bloom, 1989b). No other psychometric 
data are available in the literature. The available psychometric information should 
be interpreted with caution in that they can only serve as an estimate for this study 
because the instrument was modified. 

Quality of Wbrk Life. A modified version of the Early Childhood Work 
Environment Survey (ECWES; Jorde-Bloom, 1989a) was used to measure the 
quality of the work life perceived by the trainees at their home sites. The ECWES, 
originally designed for use with directors, measures overall organizational climate 
and staff experiences. The original instrument had 10 items. Fo' the purposes of 
this study, questions specific to director training were eliminated from the original 
ECWES. The revised version of the ECWES measured the trainee's perceptions of 
how their work environment compared with their ideal. 

There were seven items on the revised ECWES. Respondents were asked 
to rate each item (1) "strongly disagree" to (4) "strongly agree". Possible scores on 
the revised version of the ECWES range from 7 to 28. Past research has shown the 
ECWES to have high internal validity (Bloom et al., 1992). Coefficient alphas of .95 
for the pretest and .92 for the posttest have been established for the ECWES. No 
other psychometric information is available in the literature. This psychometric 
information should be interpreted as only an estimate because the ECWES was 
modified for this study. 
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The Implementation of Trainee Created Action Plans. At the end of each 
training session, the trainees submitted three specific action oriented goals based 
on the trail ;inq content. The completion of these goals was measured using an 
instrument specifically designed for this study. The instrument, the Training Goal 
Observation Scale (GOS), allowed an objective observer to rate the degree to which 
action plans were effectively implemented in the program. On a 3-point Likert-type 
scaie the observer indicated the effectiveness from "no evidence of implementation" 
(1) to "much evidence of implementation" (3) for each of the three trainee goals. The 
range of possible scores for this instrument is 3 to 9. Because this observation 
instrument was created specifically for this study, no psychometric information is 
available at this time. Inter-rater reliability was calculated to determine the accuracy 
of observations completed using the GOS. 

Goal implementation was also measured by the Trainee Goal Survey (TGS). 
This is a self-rated survey containing five items, administered in an interview setting. 
On the first three items the trainees rate the degree to which they achieved each of 
the three goals on a 3-point scale. The points on the scale range from (1) 

"did not achieve at all" to (3) '*fully achieved". On the fourth item the trainee is also 
asked to rate the ease/difficulty of implementing the goals. The points range from (1) 
difficult to (3) very easy. Finally, the trainee is asked to rate the degree to which the 
training offered by the RiCCTS assisted in the implementation of their goals. The 
points on the scale range from (1) not helpful to (3) very helpful. The purpose of 
administering an interview and employing this instrument was to collect more in- 
depth data. The interview was intended as a safeguard against missing 
programmatic changes not detected by the 
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other quantitative methods. Goal implementation was also measured by a 
supervisor version of the TGS, the S-TGS. The supervisors were asked to assess 
implementation of the trainee's goals on the same items using the same scales as 
the trainee. 

Copies of the TGS, S-TGS, and GOS are provided in Appendix C. 

Procedure 

Approximately 10 minutes prior to each training session the investigator 
distributed all of the instruments (Consent Form, Trainee Characteristics, TNAS, 
ECWES) to all of the registered trainees. The trainees were told that at the 
completion of the training they would also be asked to formulate three action based 
goals. The TNAS (S-TNAS) was mailed to all of the trainees' supervisors at that 
time. Supervisors were requested to independently complete the S-TNAS and 
return it in a postage-paid, self addressed envelope within one week. Two months 
after training, the TNAS and S-TNAS were mailed to thetraining group and their 
supervisors (S-TNAS) with each being asked to independently complete and return it 
in a postage-paid self addressed, stamped envelope. The ECWES is not completed 
by supervisors but it was mailed to the trainees. 

The two month post tests included a column on which the trainee was to 
reflect back on their competence and skill (TNAS) arxl their work setting (ECWES) 
and rate themselves according to how they were prior to training. These 
retrospective questions appeared on the Trainee form TNAS, the ECWES, and the 
Supervisor form S-TNAS. Research has shown that individuals often over-rate 
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themselves prior to training because they are not yet aware of their weaknesses until 
after the training is administered (Cozby, Worden, & Kee, 1989). For this reason, the 
trainees were given an opportunity to rate themselves on their competence, 
knowledge, and skill prior to training when completing their post test forms. 

The TNAS, ECWES, Trainee Characteristics, and consent form were mailed 
to the comparison group and their supervisors prior to their "treatment". In order to 
minimize the Hawthorne effect (Huck, Comrier, & Bounds, 1984; Adams & 
Schvaneveldt, 1991), comparison group subjects received an alternative "treatment" 
consisting of mailed materials on exercise/fitness information targeted for working 
women. This topic was chosen because it could potentially benefit the comparison 
group subjects while it is entirely unrelated to child care and any of the training 
delivered by the RICCTS. At two months after treatment, the comparison group 
subjects were also asked to complete a second TNAS and ECWES, and their 
supervsisors were asked to complete the second S-TNAS. They were provided with 
self addressed stamped envelopes in order to return their materials. 

At the completion of each training program, trainees were asked to complete 
a trainee action plan worksheet. The trainees received one copy of their goals, the 
trainer received a copy, and the evaluator received a copy. A copy of the worksheet 
is available in Appendix C. 

Six center based subjects were randomly chosen from various training 
sessions to receive an on-site observation two months after training by an objective 
observer who completed tfre GOS. At the time of the on site visit the trainees were 
given the TGS which allowed them to assess their perceived level of effectiveness in 



imiementing their goals. The supervisors received a trainee goal implementation 
rating pen-and-pencil survey and a copy of the trainee created goals. This included 
the same items that were asked for from the trainee (in interview form) during the two 
month visit. Supervisors were asked to rate the effectiveness of the trainees in 
implementing their goals, and the supervisors were asked to assess the effect the 
training had on the trainees and their program. Six home day care providers who 
attended the family day care accreditation training completed the TGS. Because 
they do not have supervisors, no S-TGS was available. 

RESULTS 

Demographics 

Hotelling's T2 were conducted to assess the compara’:ility of the training and 
comparison group for the 16 demographic characteristics. These characteristics 
included things such as race, income, education, and numbers of years in their field. 
The two groups were similar in all demographics except for two areas. The training 
group differed significantly from the comparison group in terms of type of work 
setting, F=3.506, df=33.7, p=.0001, and number of children served, F=6.322, df=96.4, 
p=.0001 . These two differences can be understood in that approximately half of the 
comparison group was home day care providers whereas only about one-fourth of 
the training group were home day care providers. Home day care providers have a 
different work setting than center based providers and serve a smaller number of 
children. 



Level of Perceived Competence 



This lesearch project investigated four major questions. The first question 
assessed changes in trainees' level of perceived competence as measured by the 
Trainee Needs Assessment Survey (TNAS). This question vyas answered by using 
the self reports of trainees and ratings provided by their supervisors. These ratings 
were obtained from trainees and supervisors prior to training and two months after 
training. Retrospective accounts of pretraining competence were also collected with 
the two month trainee and supervisor ratings. 

Change scores. Change scores were calculated and used as the dependent 
variable in the analyses. Change scores were created bysubtracting the pretest 
score (or retro) from the post-test score for each subject The mean difference score 
was used to compare the training and comparison groups in order to test whether 
one group exhibited significantly greater differences in their pre to post scores. 
Change scores were created for the TNAS and the S-TNAS. 

Supervisor and trainee agreement. Similarities in ratings provided by the 
trainees and supervisors on the TNAS were investigated by using the Pearson 
Correlation technique. The data suggests mixed results as to whether supervisors 
and trainees agree on the trainees' level of knowledge and skill. Weak positive 
correlations were found between supervisor and trainee ratings on the pretest 
r=.315, p=.136, post-test r=.375, p=.158, and retrospective accounts r=.135, p=.086. 
As can be seen on Table 1, the trainees and supervisors have different mean scores. 
Because these people have different points of view, the ratings provided by trainees 
and supervisors was considered separately in the following analyses. 
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Table 1 

Means for trainees, supervisors, comparison group and comparison group 
supervisors 



TNAS- combined 


Pre 


Post 


Retro 


Trainees 


42.939 


51 .000 


41.000 


Comparison group 


42.000 


48.383 


41 .000 


Supervisors 
Comparison sup. 


40.860 

45.375 


47.000 

47.750 


39.640 

47.375 


TNAS - personal 

professional 


Pre 


Post 


Retro 


Trainees 

Comparison group 


16.841 

18.043 


19.280 

19.174 


15.696 

19.000 


Supervisors 
Comparison sup. 


15.281 

16.625 


18.523 

17.625 


15 151 
17.250 


TNAS - child 

development 


Pre 


Post 


Retro 


Trainees 

Comparison group 


27.293 

30.130 


29.888 

29.609 


26.944 

30.217 


Supervisors 
Comparison sup. 


25.209 

28.125 


26.930 

30.125 


25.349 

30.125 


ECWES 








Trainee 

Comparison group 


21.683 

23.391 


22.872 

24.130 


20.712 

23.826 





Trainee Self Reports. A Hotelling's T2 was used to detect differences in 
change scores for training and comparison subjects for self-reported competence on 
the combined, personal / professional, and child development subscales of the 
TNAS. Two separate Hotelling's T2 were calculated. The first was completed to 
investigate potential differences between the change scores evidenced by the 
training and comparison subjects from pre to post on the three components of the 
TNAS. When compared to the comparison group, the multivariate analysis indicated 
that the training group demonstrated significant gains from pre-test to posttest 
accounts, F=23.290, df=3, p=0001. Follow up univariate analysis found significant 
pre to post irtcreases in the three areas measured by the TNAS; combined, personal/ 
professional, and child development. When compared to the comparison group, the 
training group had significant increases in their combined knowledge, t=41.725, 
df=1, p=.001, personal / professional knowledge, t=54,396, df=1, p=.001, and their 
child development knowledge, t=23.677, df=1, p=.001. The means and standard 
deviations for the change scores used in this analysis are reported on Table 2. 

The second Hotellings T2 was completed to investigate potential differertces 
between the change scores evidenced by the training and comparison subjects 
based on retrospective and post test ratings on the three components of the TNAS. 
When compared to the comparison group, the multivariate analysis indicated that the 
training group demonstrated significant gains from retrospective to posttest accounts, 
F=79.124, df=3, p=.001. Follow up univariate analyses fourxJ significant 
retrospective to post increases in the three areas measured by the TNAS; combined, 
personal professional, and child development. When compared to the comparison 
group, the training group had significant increases in their combined score. 
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t=159.473, df=1, p=.0001 . personal / professional knowledge, t=204.336, df=1, 
p=.0001, and their child development knowledge, t=47.160, df=1 , p=.0001. The 
means and standard deviations for the change scores used in this analysis are 
reported on Table 2. 

Table 2 

Means.standard deviations, and siqnificane levels of self reported change scores for 
the training and comparison groups on the TNAS. 



Training 

group 

N=125 

TNAS ore to post chanae score 


Comparison 

group 

N=23 


Significance 

level 


Scales 

combined 4.808 

(8.322) 


0.609 

(11.484) 


p=.0001 


personal 2.688 

/professional 4.076) 


1.139 

(3.334) 


p=.0001 


child 2.424 

development (5.570) 

TNAS retrospective to Dost chanae score 


-0.522 

(7.006) 


p=.0001 


Scales 

combined 6.336 

(5.610) 


-0.435 

(5.434) 


p=.0001 


personal 3.584 

/professional (2.803) 


0.174 

(2.387) 


p=.0001 


child 2.944 

development (4.793) 


-0.609 

(3.144) 


p=.0001 




Supervisor Reports. T wo separate Hotelling's T2 were calculated to analyze 
the supervisor version of the TNAS (S-TNAS). The first was completed to investigate 
potential differences between the change scores evidenced by the training and 
comparison subjects from pre to post on the three components of the S-TNAS as 
reported by the supervisors. When compared to the comparison group, the 
multivariate analysis indicated that the training group demonstrated significant gains 
from pre-test to posttest according to the supervisor reports, F=28.198, df=3, p=0001. 
Follow up univariate analysis found significant pre to post increases in the three 
areas measured by the S-TNAS; combined, personal professional, and child 
development. When compared to the comparison 1 group, the training group had 
significant increases in their combined knowledge, F=20.933, df=1 , p=0001 , 
personal / professional knowledge, F=79.025, df=1, p=.0001, and their child 
development knowledge, F=9.326, df=1 , p=.003. The means and standard 
deviations for the change scores used in this analysis ere reported on Table 2. 

The second Hotellings T2 was completed to investigate potential differences 
between the change scores evidenced by the training and comparison subjects from 
retrospective and post test reports of their supervisors on the three components of 
the S-TNAS. When compared to the control group, the multivariate analysis 
indicated that the training group had a significant Increase in total mean change 
scores from retrospective to posttest according to their supervisors, F=21 .736, df=3, 
p=.0001 . Follow up univariate analysis found significant pre to post increases in the 
three areas measured by the S-TNAS; combined, personal professional, and child 
development. When compared to the control group, the training group had 
significant increases in their combined score, t=29.798, df=1, p=.0001, personal / 
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professional knowledge, t=64.232, df=1, p=.0001, and their child development 
knowledge, t=6.714, df=1 , p=.01 1. The means and standard deviations for the gain 
scores used in this analysis are reported on Table 3. 



Table. 3 

The means, standard deviations, and significance levels of change scores for the 
training and comparison group reported by supervisors on the S-TNAS. 



Training 

group 

N=86 

S-TNAS ore to Dost mean chanae scores 


Comparison 

group 

N=8 


Significance 

level 


Scales 






combined 3.837 


2.375 


p=.0001 


(7.778) 


(3.378) 




personal 3.012 


1.000 


p=.0001 


/professional (3.142) 


(2.000) 




child 2.000 


1.721 


.p=.003 


development (5.264) 


(5.226) 




S-TNAS retro to Dost mean chanae scores 




Scales; 






combined 4.791 


0.375 


p=.0001 


(8.139) 


(0.744) 




personal 3.372 


0.375 


p=.0001 


/professional (3.902) 


(1.061) 




child 1.581 


0.000 


p=.011 


development (5.660) 


(0.535) 
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The second major question this project investigated was did the training 
change the perceived level of quality of work life for the training subjects as 
measured by the Early Childhood Work Environment Survey (ECWES). Two 
separate Hotellings T2 were calculated. The first was completed to investigate the 
potential difference between the gains evidenced between the training group and the 
comparison subjects from pre to post on the ECWES. 

Change scores. Change scores were calculated and used as the dependent 
variable in the analyses. Change scores were created bysubtracting the pretest 
score (or retro) from the post-test score for each subject The mean difference score 
was used to compare the training and comparison groups in order to test whether 
one group exhibited significantly greater differences in their pre to post scores. 
Change scores were created for the ECWES. 

When compared to the comparison group the gain score for the training 
group was significantly higher, t=23.741, df=1, p=.000. The training group evidenced 
a mean gain score of 1 .952, SD=4.479, from pre to post and the comparison group 
demonstrated a mean gain score of 0.739, SD=2.598,for pre to post. 

The second Hotellings T2 was completed to examine the potential difference 
between the gains evidenced between the training group and the comparison group 
on their self reports of pretraining retrospectively and post. When compared to the 
comparison group, the mean gain score for the training group was significantly 
higher, t=1 12.854, df=1, p=.0001. The training group evidenced a mean gain score 
of 2.160, SD=2.273, from retrospective to post and the comparison group 
demonstrated a mean gain score of 0.304, SD=0.635, from retrospective to post. 
Table 1 provides the raw scores for the ECWES. 
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Implementation of trainee created goals 

The third major question this project investigated \was the effectiveness of 
training in enabling the implementation of trainee goals. Twelve subjects were 
randomly selected, six worked in center based programs and six were home day 
care providers. The six center based subjects were randomly selected to represent 
the various workshop topics. The six home day care providers participated in the 
family day care accreditation training. Three types of information were collected 
from the center based trainees; the Trainee Goal Survey (TGS) the Supervisor 
version of the Trainee Goal Survey (S-TGS), and the Goal Observation Scale (GOS). 
The TGS was the only source of information collected from the home day care 
providers because they do not have supervisors and observations in the home care 
setting are difficult to administer. 

Trainee Goal Survey. The Trainee Goal Survey (TGS) had five items. Three 
asked the trainee to rate their perceived effectiveness in implementing each of the 
three goals. The other two questions were more general, one asked the trainee to 
rate their perceived tfie degree of difficulty in implementing the goals and the other 
questions asked the trainee to rate the degree to which the RICCTS was helpful in 

enabling the trainees to meet their goals. 

The data from the TGS were analyzed in terms of clustering subjects into 
categories based on their self reported success in goal implementation. For instance 
trainees who rated themselves as implementing all of their goals effectively (rated on 
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a scale of 1-3) were clustered into a high success category. The categories were set 
as follows, total scores 3-4 = low success, 5-7 = moderate success, 8-9 = high 
success. Of the six center based providers, none fell into the low success category, 
two were moderately successful, and four were highly successful. Out of the six 
family day care providers zero fell into the low success, one was in the moderate 
success, and five were in the high success category. The total mean score was 8.03, 
the total mean for center based child care providers was 7.97, the total mean for 
family day care providers receiving family day care accreditation training was 8.24. 

Out of the nine trainees that rated themselves in the high success category 
(five home day care, 4 center based), 100% said that their goals were "easy". One 
hundred percent said that the RICCTS was "very helpful" in enabling goal 
implementation, when asked on the TGS. Out of the three trainees who reported 
moderate success (one home day care, tv^o center based), the two center based 
providers said that their goals were "somewhat difficult' and that the RICCTS was 
"somewhat helpful". The one home day care provider who reported moderate 
success said that her goals were "easy" and that the RICCTS was "very helpfuf. 

The data from the S-TGS were analyzed by clustering subjects into similar 
categories based on supervisors’ ratings of their staffs’ competence in implementing 
the goals. Out of 6 responses, there were zero in the low category, two in the 
moderate, and four in the high success category. The tot^ mean score for the S- 
TGS was 8.13. 
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Goal Observation Survey. The six center based subjects randomly selected 
for the TGS were also visited at their center by an objective observer to administer 
the Goal Observation Survey (GOS). The GOS has three items that could be rated 
by an independent observer a scale of one to three (1= no evidence of goal 
implementation, to 3= evidence of fully implemented goal). This data were analyzed 
by clustering subjects into categories based on the score assigned by the observer. 
The observation scores were clustered into low, moderate, and high success 
categories. The categories were set as follows; 3-4 = tow, 5-7 = moderate, 8-9 = 
high. There were zero observations in the low category, there were three in the 
moderate, and three in the high category. The average mean score was 7. 

Inter-rater reliability was assessed by examining the agreement of two 
independent observers based on one observation period. The ratings were in 100% 
agreement. 

Based on the data collected by the TGS, S-TGS, and GOS the trainees were 
highly sLccessful in making desired changes. Due to the small N, analysis were not 
performed but the agreement between trainee ratings, supervisor ratings, and 
ratings obtained by an objective observer were assessed, by formulating a matrix as 
seen on Tables 4 arxl 5. As can be seen on table three, agreement was found 
between self ratings of trainee and supervisor and ratings of trainee and observer. 
This level of agreement lends confidence to these findings. 



Table 4. 

Matrix of agreement between trainee TGS scores AND supervisor S-TGS scores. 



Trainee TGS 



low moderate high 



low 0 



Supervisor moderate 2 

S>TGS 



high 



4 



Table 5. 

Matrix of agreement between trainee TGS rating and the objective observer GQS 
rating. 



Trainee TGS 



low moderate high 



low 0 



Objective moderate 2 1 

Observer 

GQS 



high 



3 
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E ffectiveness of training types 

The fourth and final major question this project investigated was potential 
differences in effectiveness of the different types of training. Three different training 
typeswere evaluated in the project: two hour workshops, eight hour conference, and 
on-going seven week training. Four separate ANOVAs were calculated each using 
different dependent measures. The dependent measures were the TNAS pre to post 
change score, the TNAS retrospective to post change score, the ECWES pre to post 
change score, and the ECWES retrospective to post change score. No significant 
differences were found in terms of training type with any of the four dependent 
measures. These analyses should be viewed as tentative given the wide variation in 
training group size. For example, the conference had 30 subjects, the workshops 
had 89, and the on-going had six. The topic, a brief description of the trainings, and 
the numbers of trainees who attended each topic are included in Appendix C. 

Discussion 

This evaluation addressed four major questions. The results produced clear 
answers to the first three questions of; (1) the training is effective in improving 
trainees' level of competence and skill? (2) it is effective in effective in improving the 
perception of work life for trainees? and (3) it is effective allowing trainee's to make 
desired changes in their programs as perceived by the trainees? 
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In regards to the fourth question, there were no significant differences in the 
effectiveness for the three types of trainings offered (one-shot workshops, 
conference, on-going). The results should be considered tentative due to the small 
number of subjects in the “on-going" group. The considerable difference in size for 
the groups poses limitations for drawing any conclusions, particularly because past 
research has found on-going training to be more effective (Sims, 1990; Jones, 

1993). 

The results of this study are strong. The results suggest that in all of the areas 
measured, the training group exhibited significant improvements. What makes these 
results particularly strong is that information was gathered from several sources of 
data (trainee, supervisor, observer), and several points in time (pre, post, and 
retrospective). In addition to using multiple measures, this study utilized a 
comparison group. 

Analysis of the data found that supervisors have different perceptions than the 
trainees, therefor their data was considered separately. These very independent 
ratings both showed that the trainees improved. In addition, an objective observer 
also found their to be changes. The agreement of all of these perspectives lends 
confidence to the firxlings. 

The use of the retrospective measure was rather unique for evaluation 
research. Research has shown that individuals often over-rate themselves prior to 
training because they are not yet aware of their weaknesses until after the training is 
administered (Cozby, Worden, & Kee, 1989). For this reason, the trainees were 
given an opportunity to rate themselves on their competence, knowledge, and skill 
prior to training when completing their post test forms. Examination of the results 



suggest that the trainees and supervisors were not aware of weaknesses until after 
training. This is based on the fact that the self rated retrospective scores on the 
instruments were lower than the pretest scores. 

A non-equivalent control group design (Cook & Campbell, 1979) was used 
inthis study. This design ircludes precautions to decrease threats to internal validity 
including history, instrumentation, pre-post sensitivity, instrumentation, and 
regression (Adams & Schvaneveldt, 1991). Threats to external validity were 
decreased as well. For example, the comparison group received a "treatment" to 
minimize the Hawthorne effect (Huck et al., 1984). 

This study is also strong because it used multiple measures. Instruments that 
were used in other evaluations were used as well as unique measures. Design and 
instrumentation warrants further discussion. . 

When conducting an evaluation two questions arise "what should be 
measured ?" and "how does one measure it?'. The question of "what to measure" 
has often been answered by examining perceived level of competerce and skill and 
examining perceived quality of work environment. With respect to "how to measure" 
these, Bloom et al. (1991) used the TNAS to assess perceived competerce and 
used the ECWES to measure quality of work life. These measures were shown to be 
effective in her evaluation of director training. 

This study utilized modified versions of the TNAS and ECWES. These were 
administered at several poii-^s in time (pre, post, and retro), and were collected from 
different sources (trainee and supervisor). Similar to previous research findings 
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(Jorde-Bloom, 1989a), these instruments were able to detect changes that can be 
attributed to training. However, the TNAS and ECWES measure perceptions. 
Perceptions do not necessarily translate into changes in behavior. Behavioral 
change is the ultimate goal of most training. Therefore, for the purposes of this study, 
goal implementation was assessed as an indicator of training effectiveness in order 
to measure changes in actual behavior. Goals allow the trainees to individualize the 
behavior they wish to change. The goal rating instruments allow an observer, in 
addition to the trainee and supervisor, to determine the effectiveness in changing 
behavior, as opposed to just perceptions as measured by the other instruments. 

The assessment of goal implementation was unique to this evaluation and 
implications can be drawn concerning the use of goals in a variety of areas related to 
child care training. These implications will be discussed in terms of training, 
professional development plans, and future research. 

Implications for training 

The use of goals yields implications for training efforts. Goals provide 
information on trainee's individuals training needs. These can be helpful to the 
RICCTS arxi other training units for planning future training. Also, a copy of the 
goals could be used by individual trainers to determine if the training they are 
delivering is being received and comprehended by their audience in a way that is 
intended. Goals also provide a tool for on-site technical assistance. If a training 
system wishes to do follow up or on-site technical assistance, goals provide a map to 
what type of help and resources each trainee would need. 
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Implications for professional development 

In addition, goals may be an effective tool for individualized professional 
development. Currently, the state mandates 20 hours of training for each child care 
provider. Goals can be used effectively to meet the individual needs of employees. 
Goals may be used as part of annual professional development plan. They can be 
used to monitor the trainees progress and to review or evaluate the trainee's 
outcome at the completion of the year. If each provider were to create individual 
goals they could become a framework for his/her professional development for the 
year. 

Implications for researeh 

Goals can be used as a measure in future evaluations. The use of goal 
implementation rating scales allows the trainee, supervisor, and obsen/er to 
determine the degree to which the trainee carries out their plans. The use of goals 
allows the evaluator to answer the question of "is the training effective" by looking at 
actual behaviors in addition to or in place of the more typically assessed changes in 
perceptions. 

Conclusions 

Several conclusions can be drawn from this study. The most obvious is that 
the training provided by the RICCTS appears to be effective and should be 
continued. A broad implication to be drawn is that training seems to be important to 
increase professionalism in the child care providers. Professionalization is certainly 
a vision for the child care community (Bloom et. al.,1991). This implication is based 
on the significant increase in trainees reports in terms of professional knowledge and 
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skill. Another broad implication is that inexpensive and available training can 
potentially improve the well being of children and should be expanded. The training 
provided by the RICCTS should be continued and expanded based on the vision of 
those involved in career development for early care providers (Morgan et. al, 1994). 

Evaluation of training is an important endeavor. Evaluation design is an area 
that needs to be further explored and expanded. Future research could utilize 
multiple sources of data and take into consideration both perceptions and behaviors. 
This v^ould add validity and reliability to the findings. Evaluation could also focus of 
what type of training is most effective and what topic areas seem to be most 
effectively trained and how. The more evaluation research is conducted the greater 
resources in the literature for those posed with the common dilemma of "what to 
measure?" and “how to measure it?" 
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